home
***
CD-ROM
|
disk
|
FTP
|
other
***
search
/
Magnum One
/
Magnum One (Mid-American Digital) (Disc Manufacturing).iso
/
d7
/
dszwords.arc
/
PCMAGTST.TXT
< prev
Wrap
Text File
|
1991-07-27
|
8KB
|
194 lines
PC Magazine Communications Software Testing Irregularities
The April 30 1991 issue of PC Magazine features reviews of 17
async communications programs that allegedly support ZMODEM and
16550A FIFO UARTs. This issue has prompted a number of messages
questioning the published test results and the procedures used
to generate them.
PC Magazine sent files from a 25 MHz 386 computer to an 8 MHz
IBM PC/AT upgraded with the 16550A UART chip. Speeds to 115200
bps were used with hardware handshaking. Software handshaking
is normally used with high speed direct computer to computer
connections, but this was disabled.
PC Magazine reported that Professional-YAM generated so many
errors and retries at 115k that throughput fell sharply.
Randol N. Tigrett, PC Magazine LAN Labs Project Leader, claims
both he and Sara Parker attempted to contact Omen Technology
about these errors. While Omen had numerous contacts with PC
Magazine at the time concerning lists of features, there is no
recall of being contacted about technical problems and/or error
messages during the performance tests. None of the FAX messages
Omen received from PC Magazine mentioned any technical problems,
and no electronic mail messages were received. Scott McGinnis,
author of COM-AND, another program that suffered from improper
PC Magazine test conditions, was not contacted either.
PC Magazine mentioned in a sidebar that Professional-YAM with
software flow control transferred files faster than all other
programs tested. PC Magazine has refused to disclose these
performance figures, either in the original article or in
response to repeated interrogatives to the test director on PC
Magazine's CompuServe bulletin board. PC Magazine's refusal to
disclose these figures is troubling: was this information
covered up because the numbers would have raised questions about
the test procedures, or because the information would have
refuted PC Magazine's speed rankings?
PC Magazine's refusal to divulge the text of the error messages
and other vital information about the tests forced me to prepare
this rebuttal.
My tests with a variety of configurations (shown below) strongly
suggest that failure to issue a "handshake on" command was the
one and only cause of Professional-YAM's failure to replicate
its first-place performance with hardware flow control. Pro-YAM
Tests (not shown here) show no speed difference between hardware
and software flow control in direct connect tests at 115 kbps.
The cause of Professional-YAM's substandard performance was
improper setup.
In my tests, Professional-YAM operated without error at 115
kbps, outperforming ProComm Plus 2.0 by at least 28 per cent. I
invite readers to repeat these tests themselves. Copies of the
test files are available on TeleGodzilla. Copies Omen's ZCOMM
shareware communications program may be used for these tests.
The ProComm Plus 2.0 transfer failures result from a ProComm bug
that disables input on 16550A/AF chips. This bug appeared in a
variety of terminal emulation and file transfer tests on a
variety of machines. Reports of similar problems at speeds as
low as 2400 bps have appeared on bulletin boards.
The purpose of this paper is not to single out ProComm for
crticism. ProComm Plus 2.0 was tested here because PC
Magazine's flawed tests ranked it the fastest.
An IBM AT modified for 8 MHz clock speed is the only
configuration I have tested that has not exhibited problems with
ProComm Plus 2.0 disabling 16550A serial input. Just replacing
the AT's CPU clock crystal with the stock IBM part causes
problems. PC Magazine has not explained their choice of such an
antique as the sole receiving test machine for tests at speeds
at least three times faster than today's fastest available
dialup modem.
The tests are, however, relevant to direct connect transfers
between PC's. (Simple cables and software flow control are the
norm for this application.) To be useful in such applications,
programs should be able to transfer entire directory trees with
a single command and maintain the most recent revision of a file
on both machines. Unfortunately, no mention is made of which
programs have these essential ZMODEM features.
A further disappointment is the failure of PC Magazine to test
ZMODEM compression with their "compressible file". The results
show, as the reader can easily verify, that ZMODEM compression
is quite effective on suitable files.
I was also disappointed that no mention was made of the programs
that violate the ZMODEM protocol description in one or more
respects.
The sine qua non of a file transfer protocol for async dial-up
applications is its performance and integrity under noisy line
conditions. How long must we wait for this critical facet to be
tested?
Since PC Magazine's prose indicates raw speed is a prime
determinant of a program's quality, it is incumbent on the
reviewers to get their facts right. As the developer of the
ZMODEM protocol, and as the author of a program that was not
given a chance to perform up to standard, I request that PC
Magazine repeat these tests under suitable supervision. If the
results or performance rankings of correctly executed tests are
different from the published values and graphics, I request that
corrected test results, graphics, and associated review comments
be properly publicized and published without undue delay.
Chuck Forsberg, Omen Technology INC April 22, 1991
----------------------------------------------------------------------------
The configurations described here duplicate PC Magazine's
published test configurations as closely as possible.
Source file: b17mh.gif, 356542 bytes, available on CompuServe
and TeleGodzilla (503-621-3746 24/96). A similarly sized
ZIPfile may be used without affecting the results. Transfers
used b17mh.gif except as noted.
Compressible file: RTTYPIX, 343851 bytes, a concatenation of
the 30 line printer prcture (RTTY art) files in my collection.
The source files chosen were long enough to allow accurate
manual timing with a stopwatch. They were stored on ramdisk.
Transfers completed without errors except as noted.
Sending machine: Micronics 33 MHz, 128k cache or Intel 386 ISBC 18 MHz
Receiving Machine: IBM 5170 PC-AT s/n00212305170 modified for 8 MHz,
replacement HD and controller (Coretest 2.7 performance index: 1.890),
CGA clone, Hayes ESP board.
Cabling: Special null modem connection with TR/DCD and RTS/CTS crossovers
Commands: speed 115200 handshake on sz -ym d:b17mh.gif
-or- speed 115200 handshake on sz -yZ d:rttypix
speed 115200 handshake on t
(ProComm set for ZMODEM auto d/l, crash recovery off)
Professional-YAM to Professional-YAM: @115kbps
50 50 50 71 kbps average
Professional-YAM to Professional-YAM 17.62: @115kbps
53 53 67 kbps
Professional-YAM to DSZ.EXE pD16384 @115 kbps
50 50 50 71 kbps av
Professional-YAM to DSZ.COM @115 kbps
82 82 83 43 kbps av
Professional-YAM to ProComm Plus 2.0: @115kbps
64 64 64 56 kbps av
As Above, to IBM PC-AT at 6 MHz (stock IBM crystal), Hayes ESP board.
Professional-YAM to Professional-YAM: @115kbps
67 67 67 53 kbps average
Compressible file: 38 38 90.5 kbps ZMODEM
Compressible file: 80 43 kbps Kermit
Professional-YAM to Professional-YAM 17.62: @115kbps
74 71 72 49 kbps av
Professional-YAM to DSZ.COM @115 kbps
112 112 32 kbps av
Professional-YAM to ProComm Plus 2.0: @115kbps
FAILED FAILED
Compressible file: FAILED
Sending to Everex System 3000 386 16 MHz 64k cache, 16550A
Professional-YAM to Professional-YAM @115kbps
35 35 102 kbps
Compressible file: 19 19 19 181 kbps ZMODEM
Compressible file: 48 72 kbps Kermit
Professional-YAM to Professional-YAM 17.62: @115kbps
48 48 74 kbps
Professional-YAM to Professional-YAM 17.62: @38.4 kbps
110 110 3241 cps
Professional-YAM to ProComm Plus 2.0: @115kbps
56 55 55 64 kbps
Compressible file FAILED FAIL ZMODEM
Compressible file 53 65 kbps Kermit
Professional-YAM to ProComm Plus 2.0: @38.4 kbps
120 2971 cps
*********************************************************
* NOTE: A 16550A/16550AF must be used!! *
*********************************************************